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ABSTRACT 

The  purpose  of  this  paper  is  to  present  the  results  of  our  reeent  experimentation  involving  a  noviee  population 
performing  aerial  door  gunnery  training  in  a  mixed  reality  simulation.  Speeifieally,  we  examined  the  effeet  that 
different  visual  displays  had  on  noviee  soldier  performanee;  qualified  infantrymen  with  maehine  gun  experienee. 
The  results  of  this  study  differed  from  the  findings  of  our  first  study,  whieh  utilized  an  expert  population  of  qualified 
helieopter  erew  members. 

The  U.S.  Army  eontinues  to  develop  new  and  effeetive  ways  to  use  simulation  for  training.  One  example  is  the  Non- 
Rated  Crew  Member  Manned  Module  (NCM3),  a  simulator  designed  to  train  helieopter  erewmembers  in  eritieal, 
high  risk  tasks.  Noviee  partieipants  were  randomly  assigned  to  one  of  two  visual  display  treatments  (fiat  sereen  or 
Head-Mounted  Display)  and  exeeuted  three  aerial  door  gunnery  training  seenarios  in  the  NCM3.  Independent 
variables  were  visual  display,  trial,  immersive  tendeney  and  simulator  siekness  questionnaire  seores.  Dependent 
variables  ineluded  performanee,  presenee  and  simulator  siekness  ehange  seores.  The  performanee  results  of  this 
study  differed  from  our  first  study  and  indieated  there  was  a  main  effeet  of  visual  display  on  performanee.  However, 
both  visual  treatment  groups  experieneed  the  same  degree  of  presenee  and  simulator  siekness.  Results  of  this  study 
indieate  that  higher  immersive  simulation  may  lead  to  better  performanee  for  a  noviee  population. 
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INTRODUCTION 
Continnation  of  Research 

The  purpose  of  this  paper  is  to  present  the  results  of  our  recent  experimentation  involving  a  novice  population 
performing  aerial  door  gunnery  training  in  a  mixed  reality  simulation.  Specifically,  we  examined  the  effect  that 
different  visual  displays  had  on  novice  soldier  performance;  defined  as  qualified  infantrymen  with  machine  gun 
experience.  The  results  of  this  study  differed  from  the  findings  of  our  first  study,  which  utilized  an  expert  population 
of  qualified  helicopter  crew  members.  This  study  is  a  continuation  of  the  research  described  in  our  previous  work 
(Stevens  &  Kincaid,  2014). 

Motivation  for  Research 

There  is  a  strong  belief  in  the  United  States  Army,  particularly  in  the  aviation  community  (Stewart,  Johnson,  & 
Howse,  2008),  that  the  greater  the  degree  of  realism  in  a  virtual  simulation,  the  more  effective  that  simulation  is. 
Similarly,  there  exists  a  strong  bias  that  the  newer  the  technology  in  the  simulator  is,  the  more  effective  that 
simulator  must  be  (Schout,  Hendrikx,  Scheele,  &  Scherpbier,  2010).  However,  little  scientific  research  exists  that 
support  these  notions  (Borgvall,  2013)  (Scales,  2013).  In  fact,  a  recent  Government  Accountability  Office  (GAO) 
report  indicated  that  the  Army  "has  not  established  metrics  or  indicators  to  assist  them  in  more  broadly  measuring 
the  impact  of  simulation-based  training  on  improving  the  performance  or  proficiency  of  servicemembers"  (United 
States  Government  Accountability  Office,  2013). 

U.S.  Army  simulation  acquisition  design  decisions  are  occasionally  made  not  based  on  sound  scientific  evidence, 
but  rather  to  satisfy  the  user  community's  wants  (Stevens  &  Kincaid,  2014).  One  such  design  trade-off,  described 
below,  and  not  grounded  in  science,  served  as  the  basis  for  this  research. 

The  Non-Rated  Crew  Member  Manned  Modnle  (NCM3) 

Crew  members,  crew  chiefs  and  flight  engineers  of  rotary  wing  aircraft  are  classified  as  non-rated  crew  members 
(NCMs)  in  Army  aviation  vernacular,  meaning  they  cannot  fly  the  aircraft.  However,  NCMs  must  perform  critical 
tasks  such  as  aerial  door  gunnery,  sling  load  operations  and  crew  coordination.  Until  very  recently  though,  these 
tasks  could  only  be  trained  in  a  live  environment  (Marton,  2008).  However,  live  training  is  costly  in  terms  of  fuel, 
ammunition  and  aircraft  maintenance  as  well  as  being  considered  a  high-risk  endeavor.  Thus  dangerous  tasks,  such 
as  aerial  door  gunnery,  are  infrequently  trained  by  crew  members  and  crew  chiefs.  To  exasperate  the  situation,  in 
OPERATION  IRAQI  FREEDOM,  the  U.S.  Army  experienced  a  shortage  of  NCMs  and  thus  had  to  employ  infantry 
soldiers  to  serve  as  door  gunners  (Curran,  2003). 

The  Non- Rated  Crew  Member  Manned  Module  (NCM3)  was  fielded  by  the  U.S.  Army  in  2011  in  order  to  train 
NCMs  in  those  critical,  high  risk  tasks.  The  NCM3  is  a  mobile,  transportable,  multi-station  virtual  simulation  device 
that  can  replicate  the  back  of  either  a  UH-60  (Blackhawk)  or  CH-47  (Chinook)  helicopter.  The  NCM3  is  capable  of 
training  aerial  door  gunnery  in  one  of  two  visual  conditions:  Head-Mounted  Display  (HMD)  or  LCD  fiat  panel 
screen. 

The  dual  visual  display  capability  of  the  NCM3  exists  as  a  result  of  a  negotiated  trade-off  conducted  between  the 
government  acquisition  office  and  the  user  community  during  the  simulation's  design  phase  (Stevens  &  Kincaid, 
2014).  The  original  design  did  not  call  for  the  use  of  any  LCD  displays  in  the  trainer.  However,  the  user  community 
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was  insistent  that  aerial  door  gunnery  must  be  capable  of  being  trained  in  a  non-HMD  mode.  Thus  a  trade-off  was 
negotiated  whereby  a  less  expensive  HMD,  with  a  narrower  field  of  view,  was  procured  and  four  LCD  flat  screen 
displays  were  installed  to  support  aerial  door  gunnery  training.  This  trade-off  resulted  in  a  training  device  that  was 
now  capable  of  supporting  mixed  reality  aerial  door  gunnery  training  via  HMD  or  LCD  display. 

This  trade-off  was  conducted  in  a  non-scientific  manner,  with  no  empirical  knowledge  of  what  the  training  effect 
would  be  and  runs  contrary  to  Department  of  Defense  modeling  and  simulation  best-practice  investment  strategies 
(Aegis  Technologies,  2008).  In  this  study,  we  examine 
performance  of  aerial  door  gunnery. 

BACKGROUND 


Milgram  &  Colquhoun's  (1999)  global  taxonomy  of 
mixed  reality  display  integration  (Figure  1)  remains  the 
taxonomy  this  study  adheres  to.  The  three  dimensions  of 
the  taxonomy  consist  of  the  Real-Virtual  continuum. 

Congruence  and  Centricity.  This  study  compares  the 
effect  on  performance  between  a  traditional  visual  display 
and  an  immersive  Head  Mounted  Display  (HMD) 
operating  in  the  red  oval  of  the  MR  Global  Taxonomy. 

Thus  the  NCM3,  as  employed  in  this  study,  is  a  mixed 
reality,  egocentric,  highly  congruent  training  simulator. 

The  goal  of  a  virtual  simulation  is  generally  to  maximize 
the  degree  of  skill  transfer  to  the  trainee,  where  transfer  is 
the  application  of  knowledge,  skills  and  abilities  acquired 
during  training  and  applied  to  the  environment  they  are 
normally  used  (Muchinsky,  2000),  (Alexander,  Brunye, 

Sidman,  &  Weil,  2005).  Transfer  has  been  empirically  demonstrated  to  occur  from  a  virtual  to  a  real  environment 
(Harrington,  2011)  (Blow,  2012)  (Seymour,  et  al.,  2002)  (Hays,  Jacobs,  Carolyn,  &  Salas,  1992)  but  this  approach 
can  be  both  difficult  and  costly.  Thus  a  more  common  approach  is  to  measure  transfer,  or  the  degree  of  learning,  in 
the  simulator  itself  This  study  employs  that  methodology. 

Fidelity  is  defined  as  the  degree  to  which  the  virtual  environment  is  indistinguishable  from  the  real  environment 
(Waller,  Hunt,  &  Knapp,  1998).  An  effective  training  simulation  need  only  possess  a  sufficient  level  of  fidelity;  the 
highest  level  of  fidelity  is  not  required  for  effective  training  (Summers,  2012)  (Thorpe,  2010)  (Borgvall,  2013) 
(Jentsch  &  Bowers,  1998).  Immersion  is  primarily  a  function  of  the  simulation's  technology  while  presence  refers  to 
the  trainee's  subjective,  personal  experience  in  the  simulation.  Similar  to  fidelity,  it  is  commonly  believed  that  more 
immersion  and  greater  presence  induced  by  the  simulator  are  correlated  with  higher  transfer.  However,  this 
relationship  is  not  clearly  supported  in  the  literature  (McMahan,  Bowman,  Zielinski,  &  Brady,  2012)  (Mikropoulos 
&  Natsis,  2011)  (Dalgamo  &  Lee,  2010)  (Persky,  et  ah,  2009)  (Selverian  &  Sung,  2003). 

Simulator  sickness  (SS)  is  defined  as  "the  unwanted  side  effects  and  aftereffects  that  may  result  from  using 
simulators,  but  does  not  result  from  similar  use  of  the  actual  equipment"  (Knerr,  2007).  SS  represents  a  major 
distraction  in  simulation  based  training  due  to  lost  training  time  and  possible  negative  habit  transfer.  HMDs  are 
becoming  more  common  in  military  virtual  simulators  but  have  been  identified  as  primary  causes  of  SS. 

Little  research  has  been  conducted  examining  the  effect  of  different  visual  solutions  in  a  virtual  simulation.  When 
experimentation  has  been  conducted,  most  empirical  results  fail  to  prove  that  a  training  benefit  exists  when 
comparing  the  use  of  HMDs  to  more  traditional  visual  displays  (Barnett  &  Taylor,  2012)  (Santos,  et  ah,  2009) 
(Knerr,  2007)  (Jacquet,  2002).  Based  on  this,  the  generally  higher  cost  of  HMDs  may  not  be  justified  due  to  the  lack 
of  a  proven  corresponding  training  benefit  when  compared  to  a  lower  cost  visual  display. 


the  effect  that  these  different  visual  displays  have  on 


Figure  1.  Global  Taxonomy  of  MR  Display  Integration 
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METHOD 

Participants 

The  Novice  group  was  composed  of  U.S.  Army  infantry  soldiers  with  prior  M240  machine  gun  experience.  Soldiers 
were  members  of  Bravo  Company,  2nd  Battalion,  124th  Infantry  Regiment,  Florida  Army  National  Guard.  The 
sample  size  of  76  soldiers  was  100%  male  and  0%  female  with  a  mean  age  of  26.2  (n  =  76,  M  =  26.2,  SD  =  6.0). 

Experimental  Objectives 

The  four  experimental  objectives  remained  the  same  from  our  prior  study,  and  are  outlined  below: 

•  The  primary  experimental  objective  (hypothesis  1)  was  to  evaluate  different  visual  displays'  effect  on 
performance  in  a  Mixed  Reality  Aerial  Door  Gunnery  Training  environment. 

•  The  second  experimental  objective  (hypothesis  2)  was  to  determine  if  different  visual  displays  resulted  in  a 
difference  in  the  level  of  simulator  sickness  experienced  in  a  Mixed  Reality  Aerial  Door  Gunnery  Training 
environment. 

•  The  third  experimental  objective  (hypothesis  3)  was  to  determine  if  there  exists  a  relationship  between  an 
individual's  immersive  tendency  score  and  their  performance  and  level  of  presence  in  a  Mixed  Reality  Aerial 
Door  Gunnery  Training  environment. 

•  The  fourth  sub-objective  (hypothesis  4)  was  to  examine  if  there  exists  a  difference  in  the  level  of  presence 
experienced  when  performing  Mixed  Reality  Aerial  Door  Gunnery  Training,  based  on  visual  display. 

Apparatus 

For  this  experiment,  one  NCM3  training  device  was  transported  to  the  unit's  location  for  the  company's  drill 
weekend.  The  primary  treatment  of  this  experiment  remained  visual  display.  Participants  either  performed  the  task 
in  stereoscopic  HMD  (NVIS  nVisor  MH60)  or  46"  flat  panel  screen  condition.  Additional  key  equipment  included 
the  use  of  the  NCM3's  demilitarized  M240  Medium  Machine  Gun  (MG)  with  simulated  recoil,  simulated  flight 
helmet,  emulated  aperture  and  InterSense  IS-900  tracking  system. 

Four  questionnaires  were  employed  for  this  second  study.  A  demographic  survey  was  used  as  the  screening 
mechanism  by  the  Principal  Investigator  to  ensure  that  all  participants  had  prior  MG  experience  and  were  not 
qualified  NCMs.  The  Simulator  Sickness  Questionnaire  (SSQ)  (Kennedy,  Lane,  Berbaum,  &  Lilienthal,  1993)  was 
used  in  this  study  to  measure  simulator  sickness  (SS).  The  SSQ  is  a  self-reporting  symptom  checklist  that  includes 
16  symptoms  associated  with  SS,  rated  on  a  four  level  scale.  The  symptom  scores  are  then  aggregated  by  subscale 
(Nausea,  Oculomotor  Discomfort,  Disorientation)  and  Total  Severity,  with  the  score  changes  used  to  determine  the 
impact  of  the  simulation  on  participants'  physiological  state.  The  Immersive  Tendencies  Questionnaire  (ITQ) 
(Witmer  &  Singer,  1998)  was  also  utilized  and  is  an  18  question  (7  point  scale)  self-reporting  checklist  that 
measures  participants'  degree  of  potential  for  immersion.  Participants'  aggregated  ITQ  score  was  used  as  a  predictor 
variable  in  this  experiment.  The  Presence  Questionnaire  (PQ)  (Witmer  &  Singer,  1998)  was  the  fourth  questionnaire 
employed  and  is  a  24  question  (7  point  scale)  self-reporting  checklist  that  measured  participants'  degree  of  presence 
experienced  during  this  experiment. 

Four  aerial  door  gunnery  scenarios,  created  with  the  assistance  of  a  subject  matter  expert  (SME),  were  again 
employed  for  this  second  study.  The  familiarization  scenario  was  developed  in  order  to  provide  participants  with  the 
opportunity  to  familiarize  themselves  with  the  simulator,  machine  gun  and  assigned  visual  treatment.  Three  formal 
scenarios  (first,  second  and  third  trial)  were  created  for  the  conduct  of  the  actual  experiment.  All  scenarios 
emphasized  kinetic  engagements,  with  the  participant's  mission  being  to  destroy  as  many  enemy  targets  as  possible 
with  the  aircraft's  machine  gun.  Scenarios  were  approximately  five  minutes  in  duration  and  contained  a  maximum  of 
50  targets  that  could  be  destroyed  by  a  soldier.  Performance  results,  defined  as  the  number  of  enemy  targets 
destroyed,  were  captured  at  the  I/O  station. 
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Experimental  Procedure 

Prior  to  starting  the  experiment,  in  mass  formation,  partieipants  were  provided  an  overview  brief  by  the  Prineipal 
Investigator  that  eovered  topies  sueh  as  the  study's  purpose,  task/eonditions/standards  as  well  as  a  mission  brief  In 
smaller  groups,  eaeh  partieipant  then  read  and  signed  their  eonsent  forms  and  were  subsequently  randomly  assigned 
to  one  of  two  visual  display  groups:  HMDs  or  flat  sereen  display.  In  groups  of  four,  partieipants  eompleted  the  three 
pre-test  surveys  and  then  exeeuted  the  familiarization  and  three  formal  seenarios  (trials  one,  two  and  three).  All 
eommands  were  given  over  the  simulator's  eommunieation  network  from  the  I/O.  Seoring  was  aeeomplished  at  the 
I/O  station.  Upon  eompletion  of  the  four  seenarios,  the  group  returned  to  the  holding  area  where  they  eompleted  the 
post-test  questionnaires. 

Data  Analysis 

This  study  employed  a  2  X  3  repeated  measures  design,  drawing  partieipants  from  a  noviee  population.  The 
independent  variables  were  display  type  (HMD  versus  LCD  flat  sereen)  and  trial  (seenarios  one,  two  and  three). 
Additional  independent  variables  were  the  partieipant's  Immersive  Tendeney  Questionnaire  (ITQ)  seore  and  initial 
SSQ  seore.  Dependent  variables  were  performanee,  final  SSQ  seores  and  Presenee  Questionnaire  (PQ)  seore. 
Display  type  was  a  between-subjeets  independent  variable.  Trial  was  a  within-subjects  independent  variable. 

Data  was  analyzed  using  a  2  (display  type)  X  3  (trial)  repeated  measures  analysis  of  varianee  (ANOVA)  for 
performanee  (number  of  enemy  targets  destroyed)  by  display  type  for  hypothesis  1.  For  hypothesis  2,  simulator 
siekness  (total  severity  and  sub-seale  aggregate  seores)  was  analyzed  by  using  separate  single-factor  ANOVAs.  A 
series  of  simple  linear  regression  tests  were  conducted  to  determine  whether  a  linear  relationship  existed  between 
the  explanatory  variable  (ITQ  Score)  and  the  response  variables,  performance  and  level  of  presence  for  hypothesis  3. 
For  hypothesis  4,  level  of  presence  was  analyzed  using  a  single-factor  ANOVA. 

RESULTS 

Hypothesis  1:  Performance  Effect  of  Different  Visual  Modalities 

Hypothesis  1  was  "the  mean  door  gunner  performance  in  the  HMD  visual  group  will  be  equal  to  the  mean  door 
gunner  performance  in  the  flat  screen  visual  group".  This  hypothesis  was  derived  from  our  literature  review,  which 
indicated  that  wearable  simulation  (HMDs)  generally  provided  little  to  no  benefit  in  performance  when  compared  to 
traditional  flat  screen  displays.  The  results  of  this  experiment  forced  us  to  reject  this  hypothesis  and  conflicted  with 
the  results  of  our  previous  study. 

ANOVA  found  a  significant  main  effect  of  visual  display  on  performance  F  (1,  222)  =  13.59,  p  =  0.00.  ANOVA 
also  did  indicate  a  significant  main  effect  of  scenario  on  performance,  F  (2,  222)  =  3.48,  p  =  0.03.  The  interaction 
between  visual  display  and  trial  was  not  significant,  F  (2,  222)  =  0.01,  =  0.99.  ANOVA  was  conducted  at  a  =  0.05. 


A  series  of  post-hoc  Student's  t-Tests  were 
performed  to  test  hypothesis  1,  employing  a 
Bonferroni  correction  (a  =  0.01).  Post-hoc  tests 
revealed  no  significant  effect  of  visual  display  on 
performance  for  scenario  1  [t(74)  =  2.27,  p  =  0.03]. 

There  was  no  significant  effect  of  visual  display  on 
performance  for  scenario  2  [t(74)  =  1.85,  p  =  0.07]. 

There  was  no  significant  effect  of  visual  display  on 
performance  for  scenario  3  [t(74)  =  2.33,  p  =  0.02]. 

However,  we  reject  this  hypothesis  as  a  lack  of 
support  has  been  found. 

There  is  a  significant  difference  in  the  mean  door 
gunner  performance  of  the  HMD  visual  group  and  the  Figure  2.  Visual  Display  Effect  ou  Performauce 
flat  screen  visual  group  at  a  =  0.05  (Figure  2).  This 
indicates  that  performance  was  affected  by  visual  display. 
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Hypothesis  2:  Level  of  Simulator  Sickness 

Hypothesis  2  was  "simulator  sickness  is  greater  in  the  HMD  visual  group  than  the  flat  screen  visual  group".  This 
hypothesis  was  derived  from  our  literature  review,  which  clearly  supported  that  HMDs  induce  a  greater  degree  of 
simulator  sickness  than  traditional  flat  screen  displays.  The  results  of  this  experiment  forced  us  to  reject  this 
hypothesis,  similar  to  the  results  of  our  previous  study. 

Each  of  the  three  Simulator  Sickness  Questionnaire 
(SSQ)  subscales  and  Total  Severity  scores  were 
obtained  before  and  after  the  participant's  exposure  to 
the  simulator  (Figure  3).  A  series  of  one-way 
ANOVAs  (a  =  0.05),  with  the  SSQ  change  scores  as 
dependent  variables  and  visual  display  as  the 
independent  variable,  were  conducted.  No  main  effect 
of  visual  display  was  found  for  the  Nausea  subscale 
[F  (1,  74)  =  0.55,  p  =  0.46].  No  main  effect  of  visual 
display  was  discovered  for  the  Oculomotor 
Discomfort  subscale  [F  (1,  74)  =  0.37,  p  =  0.55].  No 
main  effect  of  visual  display  was  discovered  for  the 
Disorientation  subscale  [F  (1,  74)  =  1.71,/)  =  0.19]. 

Finally,  no  main  effect  of  visual  display  was 
discovered  for  the  Total  Severity  score  [F  (1,  74)  = 

1.20,/)  =  0.28]. 

There  was  no  significant  effect  of  visual  display  on  simulator  sickness.  We  reject  this  hypothesis  as  no  support  has 
been  found.  We  conclude  that  there  is  no  statistical  difference  in  the  SSQ  subscale  scores  nor  Total  Severity  scores 
between  both  visual  display  groups. 

Hypothesis  3:  Level  of  Immersion 

Hypothesis  3  was  "there  does  not  exist  a  relationship  between  an  individual's  immersive  tendency  score  and  their 
performance  and  level  of  presence  in  a  Mixed  Reality  Aerial  Door  Gunnery  Training  environment" .  This  hypothesis 
was  derived  from  our  belief  that  an  individual's  tendency  to  be  immersed,  as  measured  by  the  ITQ,  had  no  effect  on 
an  individual's  performance  nor  their  level  of  presence  experienced  in  the  simulator.  The  results  of  this  experiment 
led  us  to  fail  to  reject  this  hypothesis,  as  significant  support  was  found.  This  was  congruent  with  our  prior  study's 
findings. 


The  first  regression  test  executed  was  all  participants' 
performance  coupled  with  their  ITQ  score  (Figure  4). 
Performance  was  calculated  as  the  average  score  the 
participant  achieved  over  their  three  trials.  ITQ  score 
did  not  predict  subject  performance,  P  =  -0.02,  t(75)  =  - 
0.47,  p  =  0.64.  ITQ  score  also  did  not  explain  a 
significant  proportion  of  variance  in  performance  scores 
R2  =  0.003,  F(l,  74)  =  0.22,  p  =  0.64.  Additionally, 
subsequent  simple  linear  regression  tests  revealed  that 
no  significant  linear  relationship  existed  between  an 
individual's  ITQ  score  and  performance  when  both 
visual  treatment  groups  were  examined  separately. 


Figure  4.  Liuear  Regressiou  Results  for  ITQ  aud 
Performauce  Scores 
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Figure  3.  Meau  Sickness  Questionnaire  Change  Scores 
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The  second  simple  linear  regression  test  conducted 
was  to  determine  if  a  linear  relationship  existed 
between  the  explanatory  variable  (Immersive 
Tendencies  Questionnaire  Score  (ITQ))  and  the 
response  variable,  Presence  Questionnaire  (PQ) 
score  (Figure  5).  We  found  no  linear  relationship 
exists.  ITQ  score  did  not  predict  PQ  score,  P  =  - 
0.07,  t(75)  =  -0.51,/>  =  0.61.  ITQ  score  also  did  not 
explain  a  significant  proportion  of  variance  in  PQ 
scores  =  0.003,  F(I,  74)  =  0.26,  p  =  0.61. 
Additionally,  similar  to  performance,  when 
segregated  by  visual  treatment,  we  found  no  linear 
relationship  existed  between  the  explanatory 
variable  (ITQ)  and  the  response  variable.  Presence 
Questionnaire  (PQ)  score. 
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Figure  5.  Linear  Regression  Results  for  ITQ  and  PQ  Scores 


We  fail  to  reject  hypothesis  3  as  support  has  been  found.  The  results  of  simple  linear  regression  tests  lead  us  to 
conclude  that  there  was  no  significant  linear  relationship  found  between  an  individual's  immersive  tendency  and 
their  performance,  nor  their  level  of  presence  experienced. 


Hypothesis  4:  Level  of  Presence 


Hypothesis  4  was  "the  level  of  presence  in  the  HMD  visual  group  will  he  equal  to  the  level  of  presence  in  the  flat 
screen  visual  group".  This  hypothesis  was  derived  from  our  literature  review,  which,  based  upon  recent 
experimentation,  indicated  that  the  degree  of  presence  experienced  in  simulation  can  be  independent  of  the 
simulator's  physical  immersion.  In  other  words,  higher 
presence  has  been  empirically  demonstrated  in  lower 
immersive  settings  and  vice  versa.  The  results  of  this 
experiment  led  us  to  fail  to  reject  this  hypothesis,  as 
significant  support  was  found.  This  was  congruent  with 
our  prior  study's  findings. 

Participants  completed  their  Presence  Questionnaire 
(PQ)  after  simulator  exposure.  The  PQ  provided  a 
numerical  value  of  the  degree  of  presence  the  trainee 
experienced,  as  reported  by  the  individual.  The 
independent  variable  was  visual  display.  No  main 
effect  of  visual  display  was  found  for  the  Presence 

Questionnaire  score  [F  (1,  74)  =  3.89,  p  =  0.05]  in  Figure  6.  Mean  Presence  Questionnaire  Scores 
accordance  with  Figure  6. 
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There  was  no  significant  effect  of  visual  display  on  level  of  presence  experienced  in  the  simulator.  We  conclude  that 
the  degree  of  presence  experienced  in  the  simulation  was  statistically  equal  between  both  visual  treatment  groups. 


DISCUSSION 


Conclusions 

The  test  results  of  Hypothesis  1,  "the  mean  door  gunner  performance  in  the  HMD  visual  group  will  be  equal  to  the 
mean  door  gunner  performance  in  the  flat  screen  visual  group"  indicated  a  statistically  significant  difference  in 
performance  between  the  HMD  group  and  the  LCD  group  at  a  =  .05,  with  HMD  gunners  outperforming  the  LCD 
group.  The  novice  group's  performance  also  did  not  remain  constant  amongst  consecutive  trials  at  a  =  .05.  These 
results  conflicted  with  our  prior  study  whereby  we  determined  that  expert  performance  was  not  affected  by  visual 
display. 
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A  possible  explanation  for  the  conflicting  performance  results  amongst  different  expertise  groups  involves  the 
spatial  component  of  the  aerial  door  gunnery  training  task.  In  order  to  be  successful,  participants  needed  to  rapidly 
detect  and  acquire  targets  prior  to  engaging  them  with  the  machine  gun.  Target  detection  and  acquisition,  the 
primary  spatial  component  of  this  task,  was  easier  for  participants  in  the  HMD  treatment  than  the  LCD  treatment 
because  of  both  the  slightly  wider  field  of  view  (-8°  in  each  direction)  as  well  as  the  HMD's  head-tracking 
capability.  The  HMD  group  had  precious  more  milliseconds  to  detect  and  acquire  incoming  targets  than  the  LCD 
group.  The  expert  group  was  able  to  overcome  that  time  differential,  in  large  part  due  to  their  familiarity  and 
experience  with  the  training  task.  For  the  novice  group,  who  had  no  prior  NCM  experience  but  did  possess  machine 
gun  experience,  this  proved  to  be  the  primary  driver  behind  the  performance  difference  found  between  visual  display 
groups.  This  finding  is  congruent  with  Knerr's  (2007)  determination  that  HMDs  are  slightly  more  effective  than 
traditional  visual  displays  when  performing  spatially  oriented  tasks  in  a  virtual  environment. 

The  test  results  of  Hypothesis  2,  "simulator  sickness  is  greater  in  the  HMD  visual  group  than  the  flat  screen  visual 
group"  indicated  no  difference  in  the  level  of  simulator  sickness  between  both  visual  display  groups  at  a  =  .05  and 
was  thus  rejected.  We  conclude  that  the  level  of  simulator  sickness  was  the  same  amongst  both  visual  groups.  This 
result  mirrored  our  findings  from  the  first  study.  Similar  to  the  first  study,  the  stationary  nature  of  the  task,  the 
comparable  field  of  views  afforded  by  both  visual  treatments  and  the  brief  simulator  exposure  time  all  represent 
possible  explanations  for  the  equal  level  of  SS  amongst  both  visual  treatments. 

Hypothesis  3  was  "there  does  not  exist  a  relationship  between  an  individual's  immersive  tendency  score  and  their 
performance  and  level  of  presence  in  a  Mixed  Reality  Aerial  Door  Gunnery  Training  environment ",  The  test  results 
of  hypothesis  3  indicated  no  linear  relationship  existed  between  a  subject's  Immersive  Tendencies  Questionnaire 
Score  (ITQ)  and  their  performance  score  nor  their  perceived  level  of  presence.  This  determination  is  consistent  with 
our  findings  from  the  first  study,  utilizing  an  expert  population.  Similar  to  before,  we  again  question  the  utility  of  the 
ITQ  as  a  predictive  tool. 

The  test  results  of  Hypothesis  4,  "the  level  of  presence  in  the  HMD  visual  group  will  be  equal  to  the  level  of 
presence  in  the  flat  screen  visual  group"  indicated  no  difference  in  the  level  of  presence  perceived  by  subjects  in 
either  visual  treatment  at  a  =  .05.  The  lack  of  difference  in  the  level  of  presence  between  both  visual  groups  was 
consistent  with  the  results  from  our  first  study.  Again,  we  posit  that  the  level  of  presence  experienced  by  participants 
was  driven  more  by  the  kinetic  emphasis  of  the  task  being  trained  than  the  visual  display  being  utilized. 

Lessons  Learned 

The  results  of  this  study  demonstrated  that  visual  display  had  a  significant  effect  on  novice  door  gunners' 
performance  in  a  mixed  reality  simulator.  The  novice  group's  result  was  unexpected  by  the  authors,  based  upon  both 
the  literature  review  conducted  as  well  as  the  results  from  our  first  study  utilizing  an  expert  population.  This 
experiment  would  have  provided  more  meaningful  results  if  the  participants  were  subsequently  tested  on  aerial  door 
gunnery  in  a  live  environment  (using  real  aircraft,  weaponry  and  ammunition)  after  simulator  exposure.  However, 
due  to  resource  constraints  and  safety  considerations,  this  was  not  possible. 

Additionally,  this  study  demonstrated  that  simulator  sickness  was  statistically  equal  between  the  HMD  group  and 
the  LCD  flat  screen  group,  an  unexpected  result.  However,  the  Principal  Investigator  designed  this  experiment  to 
support  throughput  and  thus  limited  participating  soldiers'  individual  exposure  to  the  simulator  to  under  30  minutes. 
This  undoubtedly  affected  the  degree  of  SS  experienced  by  participants,  per  the  literature  review.  Expectedly, 
immersive  tendency  was  not  found  to  be  a  reliable  predictor  of  performance  nor  of  presence  experienced  by  the 
participants.  Finally,  the  level  of  presence  experienced  by  participants  did  not  differ  by  visual  display  group,  as 
expected. 

This  study  can  hopefully  serve  as  a  tool  in  the  Program  Manager's  toolkit  in  order  to  have  a  professional  dialogue 
with  his/her  requirements  manager  and/or  program  proponent.  Procuring  dual  visual  solutions  for  one  simulator  not 
only  increased  the  complexity  of  the  system  but  also  unnecessarily  increased  the  cost.  While  the  cost  increase  was 
not  dramatic,  due  to  the  low  number  of  manned  modules  in  the  NCM3,  this  may  not  always  be  the  case.  This  study 
hopefully  highlighted  that  the  latest  technology  is  not  always  the  optimal  solution.  Analysis  and  trade-offs  must  be 
conducted  and  all  parties  must  have  a  baseline  understanding  of  the  second  and  third-order  effects  of  these 
decisions. 
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Furthermore,  the  results  and  lessons  learned  from  this  study  are  not  unique  to  the  military  nor  specific  to  aerial  door 
gunnery.  The  most  effective  visual  display  selection  is  of  paramount  importance  to  the  designers  of  driving,  flying, 
manufacturing  and  countless  other  types  of  virtual  simulators  involving  psychomotor  task  training.  As  examples, 
Bruzzone  et  al.  (2011)  described  their  development  of  a  prototype  portainer  virtual  simulator  that  will  be  used  to 
train  operators  in  port-related  tasks,  van  Wyck  &  de  Villiers  (2009)  outlined  their  approach  to  developing  a  virtual 
reality  prototype  for  safety  training  in  mining  and  drilling  operations.  Insinna  (2014)  described  industry's  visual 
solution  options  for  a  new  F-35  tactical  simulation.  This  study  may  have  been  of  benefit  to  those  authors  during  their 
respective  simulation  design  phases,  when  they  chose  their  visual  solutions. 

Recommendations  for  Fntnre  Research 

In  this  study,  we  analyzed  the  effect  different  visual  displays  had  on  the  performance  of  aerial  door  gunnery  in  a 
mixed  reality  training  environment.  Although  it  was  not  possible  to  validate  the  study's  results  in  a  live  training 
environment,  this  would  obviously  represent  the  major  recommendation  moving  forward.  The  measurement  of  the 
degree  of  transfer,  based  upon  visual  display  used  in  the  simulator,  to  the  live  environment,  would  be  a  worthwhile 
endeavor.  Similar  to  this  study,  the  recommendation  would  be  for  the  research  team  to  employ  a  between-subjects 
experimental  design. 

Other  recommendations  for  future  research  would  include  the  addition  of  other  visual  treatments,  such  as  a  wider 
FOV  FIMD  and  wider  FOV  adjustable  flat  screen  displays,  to  ascertain  the  true  impact  of  FOV  on  this  particular 
training  task.  Adding  an  additional,  discontinuous  session  with  the  participants  and  comparing  the  rate  of 
performance  decay  that  both  visual  display  groups  experience  would  present  another  opportunity  for  valuable 
research.  Finally,  the  possible  inclusion  of  a  third  group  to  this  study,  with  a  lower  expertise  level  than  the  current 
novice  bracket  (perhaps  from  a  combat  support  unit)  would  be  noteworthy  research  as  well. 
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